Tagging Similar Images Using Neural Network

ABSTRACT

An approach is provided in which a knowledge manager selects an extraction layer from a convolutional neural network that was trained on an initial set of images. The knowledge manager processes subsequent images obtained from crawling a computer network that includes extracting image feature sets of the subsequent images from the selected extraction layer and generating tags from metadata associated with the subsequent images. In turn, the knowledge manager receives a new image, extracts a new image feature set from the selected extraction layer, and assigns one or more of the tags to the new image based upon evaluating the new image feature set to the image features sets of the subsequent images.

BACKGROUND

The present disclosure relates to tagging similar images using aconvolutional neural network.

Deep learning is a branch of machine learning based on a set ofalgorithms that model high-level abstractions in data by using multipleprocessing layers with complex structures. Various deep learningarchitectures are used in technological systems such as computer visionsystems, automatic speech recognition systems, natural languageprocessing systems, audio recognition systems, question answer systems,etc.

The technological systems, such as question answer systems, may employclassifiers utilizing the deep learning architectures to generatedecisions for a set of input criteria. The classifiers may includealgorithms structured in the form of a deep learning architecture suchas a convolutional neural network that includes various combinations ofconvolutional layers and fully connected layers.

Prior to using classifiers in a question answer system, the classifiersare specifically trained using a substantial set of training samplessuch that the classifiers are able to classify new information receivedby the question answer system.

BRIEF SUMMARY

According to one embodiment of the present disclosure, an approach isprovided in which a knowledge manager selects an extraction layer from aconvolutional neural network that was trained on an initial set ofimages. The knowledge manager processes subsequent images obtained fromcrawling a computer network that includes extracting image feature setsof the subsequent images from the selected extraction layer andgenerating tags from metadata associated with the subsequent images. Inturn, the knowledge manager receives a new image, extracts a new imagefeature set from the selected extraction layer, and assigns one or moreof the tags to the new image based upon evaluating the new image featureset to the image features sets of the subsequent images.

The foregoing is a summary and thus contains, by necessity,simplifications, generalizations, and omissions of detail; consequently,those skilled in the art will appreciate that the summary isillustrative only and is not intended to be in any way limiting. Otheraspects, inventive features, and advantages of the present disclosure,as defined solely by the claims, will become apparent in thenon-limiting detailed description set forth below.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The present disclosure may be better understood, and its numerousobjects, features, and advantages made apparent to those skilled in theart by referencing the accompanying drawings, wherein:

FIG. 1 is a block diagram of a data processing system in which themethods described herein can be implemented; and

FIG. 2 provides an extension of the information handling systemenvironment shown in FIG. 1 to illustrate that the methods describedherein can be performed on a wide variety of information handlingsystems which operate in a networked environment;

FIG. 3 is a diagram depicting a knowledge manager that enhancesclassification ability of a classifier based upon extracting featuresets from images obtained from a computer network and using theextracted feature sets for subsequent image classification;

FIG. 4 is a diagram depicting a domain space that includes feature setsextracted from a convolutional neural network's extraction layer;

FIG. 5 is a diagram depicting a knowledge manager assigning tags to anew image based on performing a nearest neighbors search on feature setsextracted from the new image;

FIG. 6 is a flowchart showing steps taken to train a classifier forimage classification and then use extracted features from the trainedclassifier to classify images with tags obtained from crawling acomputer network; and

FIG. 7 is a flowchart showing steps taken to assign a classification toa new image based upon classifications determined in FIG. 6.

DETAILED DESCRIPTION

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the disclosure.As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present disclosure has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the disclosure in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the disclosure. Theembodiment was chosen and described in order to best explain theprinciples of the disclosure and the practical application, and toenable others of ordinary skill in the art to understand the disclosurefor various embodiments with various modifications as are suited to theparticular use contemplated.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions. The following detailed description willgenerally follow the summary of the disclosure, as set forth above,further explaining and expanding the definitions of the various aspectsand embodiments of the disclosure as necessary.

FIG. 1 depicts a schematic diagram of one illustrative embodiment of aquestion/answer (QA) system knowledge manager 100 in a computer network102. Knowledge manager 100 may include a computing device 104(comprising one or more processors and one or more memories, andpotentially any other computing device elements generally known in theart including buses, storage devices, communication interfaces, and thelike) connected to the computer network 102. The network 102 may includemultiple computing devices 104 in communication with each other and withother devices or components via one or more wired and/or wireless datacommunication links, where each communication link may comprise one ormore of wires, routers, switches, transmitters, receivers, or the like.Knowledge manager 100 and network 102 may enable question/answer (QA)generation functionality for one or more content users. Otherembodiments of knowledge manager 100 may be used with components,systems, sub-systems, and/or devices other than those that are depictedherein.

Knowledge manager 100 may be configured to receive inputs from varioussources. For example, knowledge manager 100 may receive input from thenetwork 102, a corpus of electronic documents 107 or other data, acontent creator 108, content users, and other possible sources of input.In one embodiment, some or all of the inputs to knowledge manager 100may be routed through the network 102. The various computing devices 104on the network 102 may include access points for content creators andcontent users. Some of the computing devices 104 may include devices fora database storing the corpus of data. The network 102 may include localnetwork connections and remote connections in various embodiments, suchthat knowledge manager 100 may operate in environments of any size,including local and global, e.g., the Internet. Additionally, knowledgemanager 100 serves as a front-end system that can make available avariety of knowledge extracted from or represented in documents,network-accessible sources and/or structured resource sources. In thismanner, some processes populate the knowledge manager with the knowledgemanager also including input interfaces to receive knowledge requestsand respond accordingly.

In one embodiment, the content creator creates content in a document 107for use as part of a corpus of data with knowledge manager 100. Thedocument 107 may include any file, text, article, or source of data foruse in knowledge manager 100. Content users may access knowledge manager100 via a network connection or an Internet connection to the network102, and may input questions to knowledge manager 100 that may beanswered by the content in the corpus of data. As further describedbelow, when a process evaluates a given section of a document forsemantic content, the process can use a variety of conventions to queryit from the knowledge manager. One convention is to send a well-formedquestion. Semantic content is content based on the relation betweensignifiers, such as words, phrases, signs, and symbols, and what theystand for, their denotation, or connotation. In other words, semanticcontent is content that interprets an expression, such as by usingNatural Language (NL) Processing. In one embodiment, the process sendswell-formed questions (e.g., natural language questions, etc.) to theknowledge manager. Knowledge manager 100 may interpret the question andprovide a response to the content user containing one or more answers tothe question. In some embodiments, knowledge manager 100 may provide aresponse to users in a ranked list of answers.

In some illustrative embodiments, knowledge manager 100 may be the IBMWatson™ QA system available from International Business MachinesCorporation of Armonk, N.Y., which is augmented with the mechanisms ofthe illustrative embodiments described hereafter. The IBM Watson™knowledge manager system may receive an input question which it thenparses to extract the major features of the question, that in turn arethen used to formulate queries that are applied to the corpus of data.Based on the application of the queries to the corpus of data, a set ofhypotheses, or candidate answers to the input question, are generated bylooking across the corpus of data for portions of the corpus of datathat have some potential for containing a valuable response to the inputquestion.

The IBM Watson™ QA system then performs deep analysis on the language ofthe input question and the language used in each of the portions of thecorpus of data found during the application of the queries using avariety of reasoning algorithms. There may be hundreds or even thousandsof reasoning algorithms applied, each of which performs differentanalysis, e.g., comparisons, and generates a score. For example, somereasoning algorithms may look at the matching of terms and synonymswithin the language of the input question and the found portions of thecorpus of data. Other reasoning algorithms may look at temporal orspatial features in the language, while others may evaluate the sourceof the portion of the corpus of data and evaluate its veracity.

The scores obtained from the various reasoning algorithms indicate theextent to which the potential response is inferred by the input questionbased on the specific area of focus of that reasoning algorithm. Eachresulting score is then weighted against a statistical model. Thestatistical model captures how well the reasoning algorithm performed atestablishing the inference between two similar passages for a particulardomain during the training period of the IBM Watson™ QA system. Thestatistical model may then be used to summarize a level of confidencethat the IBM Watson™ QA system has regarding the evidence that thepotential response, i.e. candidate answer, is inferred by the question.This process may be repeated for each of the candidate answers until theIBM Watson™ QA system identifies candidate answers that surface as beingsignificantly stronger than others and thus, generates a final answer,or ranked set of answers, for the input question. More information aboutthe IBM Watson™ QA system may be obtained, for example, from the IBMCorporation website, IBM Redbooks, and the like. For example,information about the IBM Watson™ QA system can be found in Yuan et al.,“Watson and Healthcare,” IBM developerWorks, 2011 and “The Era ofCognitive Systems: An Inside Look at IBM Watson and How it Works” by RobHigh, IBM Redbooks, 2012.

Types of information handling systems that can utilize knowledge manager100 range from small handheld devices, such as handheld computer/mobiletelephone 110 to large mainframe systems, such as mainframe computer170. Examples of handheld computer 110 include personal digitalassistants (PDAs), personal entertainment devices, such as MP3 players,portable televisions, and compact disc players. Other examples ofinformation handling systems include pen, or tablet, computer 120,laptop, or notebook, computer 130, personal computer system 150, andserver 160. As shown, the various information handling systems can benetworked together using computer network 100. Types of computer network102 that can be used to interconnect the various information handlingsystems include Local Area Networks (LANs), Wireless Local Area Networks(WLANs), the Internet, the Public Switched Telephone Network (PSTN),other wireless networks, and any other network topology that can be usedto interconnect the information handling systems. Many of theinformation handling systems include nonvolatile data stores, such ashard drives and/or nonvolatile memory. Some of the information handlingsystems shown in FIG. 1 depicts separate nonvolatile data stores (server160 utilizes nonvolatile data store 165, and mainframe computer 170utilizes nonvolatile data store 175. The nonvolatile data store can be acomponent that is external to the various information handling systemsor can be internal to one of the information handling systems. Anillustrative example of an information handling system showing anexemplary processor and various components commonly accessed by theprocessor is shown in FIG. 2.

FIG. 2 illustrates information handling system 200, more particularly, aprocessor and common components, which is a simplified example of acomputer system capable of performing the computing operations describedherein. Information handling system 200 includes one or more processors210 coupled to processor interface bus 212. Processor interface bus 212connects processors 210 to Northbridge 215, which is also known as theMemory Controller Hub (MCH). Northbridge 215 connects to system memory220 and provides a means for processor(s) 210 to access the systemmemory. Graphics controller 225 also connects to Northbridge 215. In oneembodiment, PCI Express bus 218 connects Northbridge 215 to graphicscontroller 225. Graphics controller 225 connects to display device 230,such as a computer monitor.

Northbridge 215 and Southbridge 235 connect to each other using bus 219.In one embodiment, the bus is a Direct Media Interface (DMI) bus thattransfers data at high speeds in each direction between Northbridge 215and Southbridge 235. In another embodiment, a Peripheral ComponentInterconnect (PCI) bus connects the Northbridge and the Southbridge.Southbridge 235, also known as the I/O Controller Hub (ICH) is a chipthat generally implements capabilities that operate at slower speedsthan the capabilities provided by the Northbridge. Southbridge 235typically provides various busses used to connect various components.These busses include, for example, PCI and PCI Express busses, an ISAbus, a System Management Bus (SMBus or SMB), and/or a Low Pin Count(LPC) bus. The LPC bus often connects low-bandwidth devices, such asboot ROM 296 and “legacy” I/O devices (using a “super I/O” chip). The“legacy” I/O devices (298) can include, for example, serial and parallelports, keyboard, mouse, and/or a floppy disk controller. The LPC busalso connects Southbridge 235 to Trusted Platform Module (TPM) 295.Other components often included in Southbridge 235 include a DirectMemory Access (DMA) controller, a Programmable Interrupt Controller(PIC), and a storage device controller, which connects Southbridge 235to nonvolatile storage device 285, such as a hard disk drive, using bus284.

ExpressCard 255 is a slot that connects hot-pluggable devices to theinformation handling system. ExpressCard 255 supports both PCI Expressand USB connectivity as it connects to Southbridge 235 using both theUniversal Serial Bus (USB) the PCI Express bus. Southbridge 235 includesUSB Controller 240 that provides USB connectivity to devices thatconnect to the USB. These devices include webcam (camera) 250, infrared(IR) receiver 248, keyboard and trackpad 244, and Bluetooth device 246,which provides for wireless personal area networks (PANs). USBController 240 also provides USB connectivity to other miscellaneous USBconnected devices 242, such as a mouse, removable nonvolatile storagedevice 245, modems, network cards, ISDN connectors, fax, printers, USBhubs, and many other types of USB connected devices. While removablenonvolatile storage device 245 is shown as a USB-connected device,removable nonvolatile storage device 245 could be connected using adifferent interface, such as a Firewire interface, etcetera.

Wireless Local Area Network (LAN) device 275 connects to Southbridge 235via the PCI or PCI Express bus 272. LAN device 275 typically implementsone of the IEEE 802.11 standards of over-the-air modulation techniquesthat all use the same protocol to wireless communicate betweeninformation handling system 200 and another computer system or device.Optical storage device 290 connects to Southbridge 235 using Serial ATA(SATA) bus 288. Serial ATA adapters and devices communicate over ahigh-speed serial link. The Serial ATA bus also connects Southbridge 235to other forms of storage devices, such as hard disk drives. Audiocircuitry 260, such as a sound card, connects to Southbridge 235 via bus258. Audio circuitry 260 also provides functionality such as audioline-in and optical digital audio in port 262, optical digital outputand headphone jack 264, internal speakers 266, and internal microphone268. Ethernet controller 270 connects to Southbridge 235 using a bus,such as the PCI or PCI Express bus. Ethernet controller 270 connectsinformation handling system 200 to a computer network, such as a LocalArea Network (LAN), the Internet, and other public and private computernetworks.

While FIG. 2 shows one information handling system, an informationhandling system may take many forms, some of which are shown in FIG. 1.For example, an information handling system may take the form of adesktop, server, portable, laptop, notebook, or other form factorcomputer or data processing system. In addition, an information handlingsystem may take other form factors such as a personal digital assistant(PDA), a gaming device, ATM machine, a portable telephone device, acommunication device or other devices that include a processor andmemory.

FIGS. 3 through 6 depict an approach directed towards open domaintagging of images, which can be executed on an information handlingsystem such as a knowledge manager. The knowledge manager selects anextraction layer from a classifier's convolutional neural networktrained to classify images using an initial training set. The knowledgemanager then “crawls” a computer network, such as the Internet, andlocates images with associated metadata. Using the located images andmetadata, the knowledge manager extracts feature sets from the selectedextraction layer, generates tags based on the image metadata, and storesthe extracted feature sets with tags in a domain space, which may or maynot be the same as the initial training set. In turn, the classifierevaluates new images and assigns tags based upon the domain space.

FIG. 3 is a diagram depicting a knowledge manager that enhancesclassification abilities of a classifier based upon extracting featuresets from images obtained from a computer network and using theextracted feature sets for subsequent image classification.

Knowledge manager 100 trains classifier 310 using initial training setimages 305 that includes, for example, 1,000 images of birds. Thetraining process includes training the various layers in convolutionalneural network 320. After the training process, convolutional neuralnetwork 320 is specifically trained for image classification.

Next, knowledge manager 100 selects extraction layer 330 to extractfeature sets. In one embodiment, convolutional neural network 320includes several layers of successive convolutions followed by a fewfully-connected non-convolutional layers. In this embodiment, knowledgemanager 100 may select the first fully connected layer after theconvolutional layers as the feature extraction layer.

Knowledge manager 100 then crawls computer network 350, such as theInternet, and obtains a multitude of images with metadata 340. Forexample, knowledge manager 100 may obtain a million images relating tovarious subjects, which may or may not be related to initial trainingset images 305. Knowledge manager 100 processes each image usingclassifier 310 and extracts feature sets from extraction layer 330.Knowledge manager also generates tag(s) based on the images' metadataand stores the extracted feature sets with tags 360 into domain store370. For example, an image of a Golden Retriever may have metadata suchas “dog” and the process stores an extracted feature set (e.g.,5800388549837854) and “dog” in domain store 370.

At this point, domain store 370 may include millions of feature set/tagentries based upon the amount of images processed while crawlingcomputer network 350. As such, knowledge manager 100 may use domainstore 370 to analyze and tag new images that include objects that werenot included in initial training set images 305 (see FIG. 4 andcorresponding text for further details).

FIG. 4 is a diagram depicting a domain space that includes feature setsextracted from a convolutional neural network's extraction layerdiscussed in FIG. 3. Domain space 400 resides in domain store 370 andincludes multiple feature sets represented by black dots that correspondto images obtained by crawling the Internet. As those skilled in the artcan appreciate, many more feature sets may exist in domain space 400,such as one million feature sets.

Domain space 400 shows that the feature sets are grouped into clusters410-495 based upon performing a nearest neighbors search or based onsimilar tags discussed in FIG. 6. The clusters may not be based on theinitial training set images, but rather the images obtained fromcrawling computer network 350.

As such, when a new image is received, knowledge manager 100 processesthe images using domain space 400 and provides tags to the new imageaccordingly (see FIGS. 5, 7, and corresponding text for furtherdetails).

FIG. 5 is a diagram depicting a knowledge manager assigning tags to anew image based on performing a nearest neighbors search on feature setsextracted from the new image. Knowledge manager 100 receives new image520 via API 500 and feeds new image 520 into classifier 310 that usesconvolutional neural network 320 to process the image. Knowledge manager100 extracts new image feature set 540 from extraction layer 330 anduses nearest neighbors analyzer 550 to perform a nearest neighborssearch on domain space 400 included in domain store 370.

Nearest neighbors analyzer 550 determines tags 560 based on the nearestneighbors search. In one embodiment, nearest neighbors analyzer 550performs a voting/averaging step of classes identified in the nearestneighbors search (see FIG. 7 and corresponding text for furtherdetails).

FIG. 6 is a flowchart showing steps taken by a process to train aclassifier for image classification and then use extracted features fromthe trained classifier to classify images with tags obtained fromcrawling the Internet. FIG. 6 processing commences at 600 whereupon, atstep 610, the process trains the classifier's convolutional neuralnetwork 320 using a set of initial training images 305. For example,initial training set images 305 may include 1,000 images of differentanimals.

At step 620, the process selects a specific feature extraction layer inconvolutional neural network 320 from which to extract feature setinformation. In one embodiment, convolutional neural network 320includes several layers of successive convolutions followed by a fewfully-connected non-convolutional layers. In this embodiment, theprocess may select the first fully connected layer after theconvolutional layers as the feature extraction layer. At step 630, theprocess extracts a feature set from the selected feature extractionlayer. Continuing with the embodiment above, the process extracts thefeature set from the output of the first fully connected layer.

At step 640, the process crawls the Internet via computer network 350and processes multiple images that include corresponding metadata. Forexample, the process may process a million images found on the Internet.For each image, the process 1) extracts a feature set from the selectedfeature extraction layer, 2) generates tag(s) based on the metadataassociated with the image, and 3) stores the extracted feature set andtags into a domain space that is stored in domain store 370. Forexample, an image of a Golden Retriever may have metadata such as “dog”and the process stores an extracted feature set (e.g., 5800388549837854)and “dog” in domain store 370. At this point, domain store 370 mayinclude millions of feature set/tag entries based upon the amount ofimages processed while crawling the Internet.

At step 660, the process groups the feature sets into clusters basedupon, for example, matching tag information or nearest neighbors searchresults (see FIG. 4 and corresponding text for further details). FIG. 6processing thereafter ends at 695.

FIG. 7 is a flowchart showing steps taken to assign a classification toa new image based upon classifications determined in FIG. 6. Processingcommences at 700 whereupon, at step 710, the process receives new image520. At step 720, the process processes the image with convolutionalneural network 320 and extracts a new image feature set from the featureextraction layer, which is the same feature set extraction layerselected at step 620 in FIG. 6.

At step 730, the process performs a nearest neighbors search on the newimage feature set against the extracted feature sets that wereclassified in domain store 370 at step 660 in FIG. 6. In one embodiment,the process identifies the nearest clusters and then finds the nearestimages (feature sets) within the nearest clusters.

At step 740, the process determines tags for the new image based uponthe nearest neighbors search. In one embodiment, the process performs avoting/averaging of the nearest neighbors search results and determinestags accordingly. For example, the nearest neighbors search may resultin a probability of 70% that the image includes a dog and, in turn, theprocess assigns a “dog” tag to the image.

At step 750, the process assigns the determined tags to the image andprovides the tags to a user. FIG. 7 processing thereafter ends at 795.

While particular embodiments of the present disclosure have been shownand described, it will be obvious to those skilled in the art that,based upon the teachings herein, that changes and modifications may bemade without departing from this disclosure and its broader aspects.Therefore, the appended claims are to encompass within their scope allsuch changes and modifications as are within the true spirit and scopeof this disclosure. Furthermore, it is to be understood that thedisclosure is solely defined by the appended claims. It will beunderstood by those with skill in the art that if a specific number ofan introduced claim element is intended, such intent will be explicitlyrecited in the claim, and in the absence of such recitation no suchlimitation is present. For non-limiting example, as an aid tounderstanding, the following appended claims contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimelements. However, the use of such phrases should not be construed toimply that the introduction of a claim element by the indefinitearticles “a” or “an” limits any particular claim containing suchintroduced claim element to disclosures containing only one suchelement, even when the same claim includes the introductory phrases “oneor more” or “at least one” and indefinite articles such as “a” or “an”;the same holds true for the use in the claims of definite articles.

1. A method implemented by an information handling system that includesa memory and a processor, the method comprising: training a classifieron a first set of images, wherein the classifier comprises aconvolutional neural network that includes a plurality of layers;processing a second set of images using the trained classifier, whereinthe processing comprises: extracting a plurality of feature sets from aselected layer included in the plurality of layers; and generating aplurality of tags based upon metadata corresponding to the second set ofimages; selecting a subset of the plurality of tags based on performinga nearest neighbors search on a new image feature set corresponding to anew image, wherein the new image feature set is extracted from theselected layer during processing of the new image; and assigning atleast one of the subset of the plurality of tags to the new image basedupon voting on the subset of the plurality of tags.
 2. The method ofclaim 1 wherein the selected layer is not a last layer in the pluralityof layers.
 3. The method of claim 2 wherein the plurality of layersincludes a set of convolutional layers and a subsequent set of fullyconnected non-convolutional layers, and wherein the selected layer is afirst layer in the subsequent set of fully-connected non-convolutionallayers.
 4. (canceled)
 5. (canceled)
 6. The method of claim 1 wherein thefirst set of images includes a first set of object types that aredifferent from a second set of object types included in the second setof images.
 7. The method of claim 1 further comprising searching theInternet to obtain the second set of images.
 8. An information handlingsystem comprising: one or more processors; a memory coupled to at leastone of the processors; and a set of computer program instructions storedin the memory and executed by at least one of the processors in order toperform actions of: training a classifier on a first set of images,wherein the classifier comprises a convolutional neural network thatincludes a plurality of layers; processing a second set of images usingthe trained classifier, wherein the processing comprises: extracting aplurality of feature sets from a selected layer included in theplurality of layers; and generating a plurality of tags based uponmetadata corresponding to the second set of images; selecting a subsetof the plurality of tags based on performing a nearest neighbors searchon a new image feature set corresponding to a new image, wherein the newimage feature set is extracted from the selected layer during processingof the new image; and assigning at least one of the subset of theplurality of tags to the new image based upon voting on the subset ofthe plurality of tags.
 9. The information handling system of claim 8wherein the selected layer is not a last layer in the plurality oflayers.
 10. The information handling system of claim 9 wherein theplurality of layers includes a set of convolutional layers and asubsequent set of fully connected non-convolutional layers, and whereinthe selected layer is a first layer in the subsequent set offully-connected non-convolutional layers.
 11. (canceled)
 12. (canceled)13. The information handling system of claim 8 wherein the first set ofimages includes a first set of object types that are different from asecond set of object types included in the second set of images.
 14. Theinformation handling system of claim 8 wherein at least one of the oneor more processors perform additional actions comprising: searching theInternet to obtain the second set of images.
 15. A computer programproduct stored in a computer readable storage medium, comprisingcomputer program code that, when executed by an information handlingsystem, causes the information handling system to perform actionscomprising: training a classifier on a first set of images, wherein theclassifier comprises a convolutional neural network that includes aplurality of layers; processing a second set of images using the trainedclassifier, wherein the processing comprises: extracting a plurality offeature sets from a selected layer included in the plurality of layers;and generating a plurality of tags based upon metadata corresponding tothe second set of images; selecting a subset of the plurality of tagsbased on performing a nearest neighbors search on a new image featureset corresponding to a new image, wherein the new image feature set isextracted from the selected layer during processing of the new image;and assigning at least one of the subset of the plurality of tags to thenew image based upon voting on the subset of the plurality of tags. 16.The computer program product of claim 15 wherein the selected layer isnot a last layer in the plurality of layers.
 17. The computer programproduct of claim 16 wherein the plurality of layers includes a set ofconvolutional layers and a subsequent set of fully connectednon-convolutional layers, and wherein the selected layer is a firstlayer in the subsequent set of fully-connected non-convolutional layers.18. (canceled)
 19. (canceled)
 20. The computer program product of claim15 wherein the first set of images includes a first set of object typesthat are different from a second set of object types included in thesecond set of images.